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Executive summary 



A variety of formal benchmarking 
practices are being used in higher 
education today, and these efforts 
seem to be growing in sophistica- 
tion. This report reviews those 
benchmarking practices, particularly those being 
used at community colleges, and introduces the 
concept of a "culture of inquiry" as a means for 
judging their potential value. 

Campus benchmarking 
activities share some basic 
traits: All seek to assess a 
college's achievements, 
shortcomings and environ- 
ments (often in comparison 
with peer institutions) and 
to identify strategies for 
improvement and innovation. 

This report classifies bench- 
marking efforts among three 
types — performance, diagnostic and process 
— and characterizes each by its typical use. The 
report also attempts to gauge the practical value 
of these various activities in order to judge their 
capacity to truly inform understanding of institu- 
tional productivity and effectiveness. 



Too often, accountability policies require 
institutions to report data that are never actu- 
ally used to guide decisions at the institutional or 
state levels. Because of this lost opportunity, the 
value of these efforts is often more symbolic than 
practical. To address this problem, the report says, 
"data-based decision-making" strategies must view 
campus-based practitioners — including academic 
and student services adminis- 
trators, institutional research- 
ers, faculty members and 
senior leaders — not only as 
decision makers, but also as 
potential agents of change. 

A recent Lumina Founda- 
tion report by Thomas Bailey 
and Mariana Alfonso ( Paths 
to Persistence, January 2005) 
underscored the importance 
of data-driven decision mak- 
ing, calling for institutions to create and maintain 
a "culture of evidence." Building on that idea, the 
author of this report suggests that the goal should 
instead be a culture of inquiry, one in which data 
move out of the limelight, and practitioners move 
to center stage. The report emphasizes that the 



Too often, accountability 
policies require institutions 
to report data that are 
never actually used. 
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task of creating knowledge from data is best under- 
stood as a craft, and that the practitioner's role as 
craftsperson is critical. 

As sociologist-author Richard Alford observes 
in The Craft of Inquiry-. "Evidence never contains its 
own explanation." This is a caution, not only that 
data require analysis to convey meaning, but also 
that the very process of gathering and analyzing 
data is subjective. To exercise craftsmanship in the 
inquiry process, we must recognize that each of us 
brings personal perspectives and experiences to the 
interpretation of evidence. We must understand 
that how we decide what information to collect, 
whom to involve in data interpretation, and how 
to communicate results can be as important as the 
results themselves. This is the essential difference 
between a culture of evidence and a culture of 
inquiry: The emphasis shifts from the data to the 
decision-maker as the locus of change. 

If peer comparison processes are to spur inno- 
vation and improve student success, the results of 
these comparisons must inform — and sometimes 
change — the thinking and behavior of instructors 
and administrators. Clearly, this fact has implica- 
tions for professional development in community 
colleges. In a culture of inquiry that seeks to en- 
hance student success, college administrators and 
faculty must do the following: 

Work to identify and address problems by 
purposefully analyzing data about student 
learning and progress. 

Engage in sustained professional develop- 
ment and dialogue about the barriers to 
student achievement. 

Have the capacity for insightful questioning 
of evidence and informed interpretation of 
results. 

To support the development of cultures of 
inquiry in community colleges, this report reviews 
a variety of activities that seek to assess the perfor- 
mance of these colleges in promoting student suc- 
cess. Again, these activities fall into one of three 
different categories: performance benchmarking, 



diagnostic benchmarking and process benchmark- 
ing. A description of each type follows: 

Performance benchmarking, also called 
metric benchmarking, is simplest and takes 
place through the straightforward though 
often superficial comparison of perfor- 
mance data. This report provides a snapshot 
of accountability objectives and perfor- 
mance indicators in three New England 
states to demonstrate how performance 
benchmarking is typically used at the state 
level. The National Community College 
Benchmark Project (NCCBP), a voluntary 
effort by participating colleges, dem- 
onstrates a more sophisticated use of out- 
come indicators within a conceptual system 
— one that accounts for resource inputs 
and the different educational environments 
in which those resources are expended. 

Diagnostic benchmarking is a "health 
check" that seeks to characterize an 
organization's performance status and to 
identify areas that need improvement. The 
report examines this type of benchmarking 
by discussing several well-known national 
assessment efforts. These efforts employ 
surveys of community college student ex- 
periences, behaviors and attitudes — 
surveys that use educational research and 
theories as their diagnostic framework. Un- 
like performance benchmarking systems 
that tend to treat educational experiences 
as assembly-line processes, diagnostic sys- 
tems see the student-college relationship as 
an interactive process — one that has a 
direct effect on student outcomes. Students 
are not passive "widgets" processed in an 
educational factory,- they themselves con- 
tribute to their own outcomes. 

The student role is characterized by the Com- 
munity College Student Experience Questionnaire 
(CCSEQ) in terms of "effort" and by the Com- 
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munity College Survey of Student Engagement 
(CCSSE) in terms of "engagement." The Coopera- 
tive Institute Research Program (CIRP) focuses 
on broad indicators of students' experiences in 
college, as well as their precollegiate expectations. 
These diagnostic frameworks help administrators 
focus their attention on how to improve collegiate 
practice. 

A different type of diagnostic benchmarking 
strategy is used in a project called Equity for All: 
Institutional Responsibility for Student Success. 
This project provides a 
framework for institutional 
self-assessment in which 
teams of administrators, 
faculty and institutional re- 
searchers begin by examining 
student outcome data disag- 
gregated by race and ethnic- 
ity. These inquiry teams then 
collaboratively interpret the 
meaning of the data and use 
these interpretations to deter- 
mine how to address observed inequities in student 
outcomes. 

Process benchmarking is the most 
expensive and time-consuming type 
of benchmarking. It involves an in-depth, 
comparative examination of a specific core 
practice at two or more institutions. Process 
benchmarking has been envisioned by edu- 
cational researchers, who have begun to 
argue for systematic "quasi-experiments" 
designed to isolate the characteristics of 
teaching and learning systems. Because 
the federal government favors this strategy 
and has incorporated it into definitions of 
"rigorous" program evaluation, community 
college personnel can expect more 



invitations to participate in these types of 
experiments. 

These experiments will be defined in terms 
of three critical features: (1) outcome measures,- 
(2) descriptions of the optimal academic "treat- 
ment" conditions, including the tasks undertaken 
and instructional media employed in instruction,- 
and (3) descriptions of the optimal teaching strate- 
gies associated with those tasks and technologies 
adopted to produce the specified outcomes. Such 
experiments — designed 
to be in-depth and precise 
— are a form of process 
benchmarking because they 
allow the participating or- 
ganizations to observe how 
various methods of teach- 
ing or student service affect 
student outcomes. 

This review shows that 
administrators and faculty 
members have many options 
for benchmarking activities on their campuses 
— some of them mandated by state and federal 
governments. This report should help inform the 
development of these mandates because it explains 
the broad array of current benchmarking practices 
and provides a common language with which to 
distinguish their purposes. Benchmarking activities 
will most benefit those who understand the under- 
lying theoretical frameworks and statistical foun- 
dations of the various approaches and can interpret 
results in the appropriate context. Clearly, this 
underscores the need for effective and ongoing 
professional development for community college 
practitioners. The report discusses a number of 
useful initiatives and resources designed to support 
this ongoing professional development. 



Administrators and faculty 
members have many 
options for benchmarking 
activities on their campuses. 
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Introduction 



C ommunity colleges have been de- 
scribed as experiencing a "perfect 
storm" of increasing enrollment 
pressures, declining public revenues, 
and unprecedented competition from 
the for-profit sector for students and public funds 
(Boggs, 2004). Another storm is brewing as legisla- 
tures and postsecondary as- 
sociations continue a heated 
debate about the means by 
which colleges and universi- 
ties should be held account- 
able for educating their 
students (Accountability for 
Better Results, 2005,- Bollag, 

2004,- Burd, 2004,- Fleming, 

2004,- Strout, 2004). Higher 
education's assessment move- 
ment is a countervailing force 
to the external pressures of 
accountability (Ewell, 1991,- Moore, 2002), but the 
internal focus of assessment does little to appease 
the demand for public information about college 
productivity and effectiveness. 

In a provocative "challenge essay" featured in a 
recent report issued by the Education Commission 



of the States and the League for Innovation in the 
Community College, Kay McClenney argues that 
community college educators should ask them- 
selves "hard questions" about student attainment to 
examine whether they are doing enough to ensure 
student success (McClenney, 2004, p. 11). "The 
urgent priority for [community colleges] is to be 
involved in shaping account- 
ability systems so that they 
are appropriate to community 
college missions and students, 
and so that they serve rather 
than thwart the access and 
attainment promises," she 
writes (p. 13). McClenney, 
director of the Community 
College Survey of Student 
Engagement, maps a strategy 
to reach that goal, includ- 
ing emphases on building 
connections with secondary schools, provid- 
ing effective remedial education, strengthening 
student engagement, and exercising transforma- 
tional leadership. She also calls for a "new culture 
of evidence" (p. 14) in which questions about 
"student progress, student attainment, and student 



The assessment movement 
is a countervailing force to 
external pressures 
of accountability. 
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success" are answered on campuses through careful 
data analysis. (See Page 6: "What is a culture of 
evidence?") 

Community college administrators and institu- 
tional researchers who are participants in a Think 
Tank (see Appendix A, Page 36) facilitated by the 
New England Resource Center for Higher Educa- 
tion (NERCHE) have been discussing a related 
concept: the emergence from the accountability 
and assessment movements of a "culture of inquiry" 
(Creating a Culture of Inc/uiry, 2005). In the area of 
student success, a culture of inquiry is character- 
ized by the professionalism of administrators and 
faculty who identify and address problems through 
purposeful analysis of data about student learn- 
ing and progress. A culture of inquiry depends on 
the dispositions and behaviors of the people who 
teach in and administer programs at colleges. It 
requires their willingness to engage in sustained 
professional development and dialogue about 
the barriers to student achievement. A culture 
of inquiry depends on the capacity for insightful 
questioning of evidence and informed interpreta- 
tion of results. 

To help support the development of cultures 
of inquiry in community colleges, this report 
reviews a variety of higher education activities that 
attempt to assess the performance of colleges in 
promoting student success. (See Pages 20 and 21: 
"Assessment in the learning college.") Each is, in 
one way or another, a form of benchmarking. Why 
examine benchmarking practices? Higher educa- 
tion benchmarking activities, which are designed 
to compare performance at one college with other 
colleges or against a set of performance criteria, 



were spurred by legislative accountability initia- 
tives, but they have evolved over time in response 
to educators' objections to simplistic indicators of 
college performance. Under tight budgets, state 
accountability programs have reduced the ties 
between college performance and funding and in- 
creased reliance on public reporting of results as a 
lever to promote institutional effectiveness (Burke 
& Minassians, 2003). The increased emphasis on 
information gathering, analysis and public report- 
ing has been accompanied by a growing sophisti- 
cation of administrative practices and options to 
measure performance. This means practitioners 
face a greater demand for professional knowl- 
edge and development in order to make informed 
choices in a complex arena. Benchmarking is not 
an essential component of a culture of inquiry, but 
many current assessment activities incorporate 
benchmarking strategies in one form or another. 

The review of performance-measurement 
activities presented in this report is intended to 
provide needed context for making decisions in 
this increasingly high-pressure environment. This 
report draws on a series of discussions with mem- 
bers of a national advisory council formed to assist 
in an ongoing effort to enhance the success of 
students in the nation's community colleges. (See 
Appendix B, Page 37, for a listing of the members 
of the advisory council.) The report also draws on 
a lengthier paper presented at a symposium in Fall 
2004 at Roxbury Community College in Boston, 
Mass. (Dowd & Tong, 2004). That paper is avail- 
able at: http://www.faculty.umb.edu/alicia_dowd/ 
ccssp/working_reports.htm. 
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What is a culture of evidence? 



s part of the Achieving the Dream: Community Colleges Count initiative funded by 
Lumina Foundation for Education, Thomas Bailey and Mariana Alfonso (2005) have 
characterized a "culture of evidence" as one in which "institutional research functions 
play a more prominent role and faculty and administrators are more fully engaged with data 
and research about success of their students, using those data to make decisions" (p. 3). 

The researchers provide suggestions for colleges wishing to develop a culture of evidence. A 
partial list of those suggestions includes: 

• Assess and invest in the resources and skills needed for effective institutional 
research. 

• Recognize that assessments of program effectiveness are difficult. 

• Combine qualitative and quantitative research on student outcomes. 

• Provide more opportunities for faculty and administrators to engage in the 
research process. 

• Develop more systematic methods to publicize and disseminate research findings. 

• Promote collaboration among institutional researchers at different colleges and 
between college and system research offices. 

The report, which seeks to contribute to a "continuous conversation within and among the 
colleges, and with outside researchers and policy-makers" (p. 4), is available on the Lumina 
Foundation Web site at: http:/ / www.luminafoundation.org/ publications/ PathstoPersistence.pdf. 
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What is benchmarking? 



B enchmarking is essentially a process of 

comparison for purposes of assessment or 
innovation (Bender & Schuh, 2002). The 
objective typically is for an organiza- 
tion to understand its own activities, 
achievements and shortcomings through compari- 
son with "peers." The peer group may be selected 
based on similar objective characteristics, such as 
enrollment size, or by the use of perceived best 
practices that are to provide a model for improved 
performance (Hurley, 2002). Benchmarking takes 
several forms, and a number 
of classification systems 
exist to differentiate them. 

Yarrow and Prabhu (cited in 
Doerfel & Ruben, 2002) define 
performance, diagnostic and 
process benchmarking in a manner that is relevant 
to higher education. 

Performance benchmarking (also called metric 
benchmarking) is simplest and takes place through 
the straightforward comparison of performance 
data. This approach focuses "only on superficial 
manifestations of business practices" (Doerfel & 
Ruben, 2002, p. 6). Diagnostic benchmarking is 
characterized as a "health check" intended to assess 



an organization's performance status and identify 
practices that need improvement. The third ap- 
proach, process benchmarking , is the most expensive 
and time consuming. It brings two or more organi- 
zations into an in-depth comparative examination 
of a specific core practice. 

As discussed below, accountability systems 
have relied primarily on performance (metric) 
benchmarking. Elements of diagnostic benchmark- 
ing are emerging as accountability systems mature. 
Process benchmarking has been envisioned for 

higher education by research- 
ers, who now argue for sys- 
tematic "quasi-experiments" 
designed to isolate the char- 
acteristics of effective teach- 
ing and learning systems. This 
approach is consistent with the federal govern- 
ment's push for experimental designs to be used in 
federally funded evaluations of social and educa- 
tional programs ( Rigorous Evidence , Scientifically-Based 
Research). Given this emerging emphasis, more 
and more practitioners are likely to be invited in 
coming years to participate in national evaluation 
projects designed to compare the effectiveness of 
various aspects of federally funded programs. 



Benchmarking takes 
several forms. 
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The new role of institutional research 



n he role of institutional researchers in community colleges has recently attracted a great 
deal of interest. Covering an event at the "Innovations 2005" conference of the League 
for Innovation, The Chronicle of Higher Education reported: "Community Colleges Should 
Rely More on Institutional Research, Conference Panelists Say" (Evelyn, 2005). A report of 
the National Commission on Accountability in Higher Education that advocated "explicit" and 
"demanding" standards for institutional research garnered similar attention (Fischer, 2005). 

Drawing on a study conducted by the Community College Research Center at Teachers 
College, Columbia University, panelists at the League conference noted that many community 
colleges have no full-time institutional research (IR) position and that most IR departments 
are relatively young, having been created in the past decade to respond to accountability 
and reporting requirements. The panelists, including Robert Grabinger, dean of research 
and planning at City College of San Francisco, argued that IR offices should spend less 
time on "college management," such as preparing reports for accreditation visits, and more 
time providing leadership to "help colleges fulfill their mission." In particular, a "direct 
relationship" with the college faculty was advocated in order to bring professors and 
researchers together to conduct assessments of student learning. 

In an essay published in the Journal of Applied Research in the Community College, Dan Walleri 
(2003), director of research and planning at Mt. Hood Community College in Oregon, 
describes the historical foundations of the IR role at community colleges and traces the 
evolution of the role. The essay provides a valuable characterization of the multiple functions 
played by IR offices, such as federal and state reporting, enrollment analysis and forecasting, 
compilation of college fact books, strategic planning, assessment and accreditation. In addition, 
the complexity of professional roles is highlighted through discussion of the expectations on 
researchers to navigate in political arenas and to act as the credible and objective "conscience" 
of the institution. Walleri's essay provides essential context for those seeking to reshape or 
capitalize on the institutional research function to foster student educational attainment. 
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Performance benchmarking 



T hrough the use of performance indica- 
tors, state-mandated accountability 
systems have emphasized performance 
benchmarking (Barak & Kniker, 2002). 
Nationally, the most common indica- 
tors of student success for community colleges 
have been retention, transfer, graduation and job 
placement rates (Burke & Associates, 2002,- Burke 
& Minassians, 2003). During 
the past two decades, states 
have been attempting, with 
uneven and unpromising 
results, to create funding 
systems that will prompt im- 
proved institutional perfor- 
mance. Joseph Burke, direc- 
tor of the higher education 
program at the Rockefeller 
Institute of Government at 
the State University of New 
York in Albany, has been tracking state perfor- 
mance funding initiatives since 1997 (see reports 
available at: http://www.rockinst.org/publications/ 
education.html). The results of the seventh annual 
survey conducted by Burke and his colleagues of 
State Higher Education Finance Officers (SHEFO) 



demonstrate that it has been difficult to implement 
the political, sometimes merely rhetorical goal of 
changing a funding system based on inputs (such 
as enrollment) to one based on outputs (graduation 
rates). 

Particularly during the most recent budget 
crises, several states have canceled or suspended 
performance initiatives tied to budgeting or 

funding, while others have 
diminished their expecta- 
tions of adopting such a plan. 
Survey results indicate that 
the perceived impact of these 
programs on performance has 
declined and is frequently 
rated as minimal or moderate. 
In 2003, 46 states required 
performance reporting, but 
Burke and his colleagues not- 
ed the "modest" use of such 
reports for planning, policy-making or decision 
making. They described reporting requirements as 
"symbolic policies," which "appear to address prob- 
lems, while having little substantive effect" (Burke 
& Minassians, 2003, p. 14). 



Several states have canceled 
or suspended performance 
initiatives tied to 
budgeting or funding. 
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The National Community 
College Benchmark Project 

Among performance benchmarking efforts, the 
National Community College Benchmark Project 
(NCCBP) (www.nccbp.org), led by Jeffrey Seybert 
at Johnson County Community College in Kansas, 
is the most sophisticated system. The project is a 
voluntary effort by a group of community colleges 
that has grown from 20 to more than 150 in the 
past three years. In three state systems — Ten- 
nessee, Pennsylvania and SUNY in New York 
— colleges have enrolled together in the project 
to enable statewide compari- 
sons on NCCBP indicators. 

The NCCBP indicators rep- 
resent a systemic relationship 
between inputs from the col- 
lege community and student 
outcomes. The project also 
seeks to compare the level of 
classroom resources among 
participating colleges and to 
observe fine-grained student 
outcomes. Table 1 (Page 
25) provides a listing of NCCBP indicators. (For 
complete information, see the glossary of bench- 
marks at the project's Web site: http://www.nccbp. 
org/_FileLibrary/FileImage/Glossary2005.pdf.) 

The failure of performance accountability plans 
to consider differences in student preparation, 
motivation and aspirations often generates strong 
objections from practitioners. It is not surpris- 
ing, then, that the NCCBP, which is advised by a 
knowledgeable and experienced board of practitio- 
ners, includes measures of student satisfaction and 
goal attainment along with more typical state-level 
indicators of persistence, degree completion and 
transfer. As community college practitioners often 
point out, not all students wish to attain degrees or 
certificates. 

The NCCBP indicators also account for other 
forms of diversity among community college 
students. Student progress through developmen- 
tal courses is recognized, as is the performance of 
transfer students in the four-year sector. The occu- 



pational training function is recognized by track- 
ing the employment status of former students, and 
by including employers' satisfaction rankings of 
students trained at the college. The participation 
rates of students from minority groups tradition- 
ally underrepresented in higher education are com- 
pared to the minority population in the college's 
service area. 

In addition, the NCCBP includes a series 
of input indicators that give a sense of relative 
resources available to produce outcomes. These 
include class size, student-teacher ratios and train- 
ing expenditures. The project 
also enables peer group 
selection to take account of 
differences in community 
wealth by recording service 
area unemployment rates and 
median income. Colleges 
can also select peers with 
operating budgets of similar 
size. Since participation in 
the NCCBP is voluntary, 
the colleges can select their 
peer group criteria based on their decision-mak- 
ing and strategic-planning needs. This differs from 
peer group creation processes for accountability 
purposes, where colleges may find themselves 
assigned through politically sensitive processes 
to groups that may or may not be appropriate for 
informative performance comparisons. (See Page 
1 1 : "Selecting peer groups: An art and a science.") 

Performance reporting 
for state accountability 

A closer look at accountability objectives and 
performance reporting requirements in three New 
England states reflects national trends and illus- 
trates the current status of such policies. Table 2, 
FJigher Education Accountability Objectives and 
Performance Indicators (Pages 26 and 27), presents 
information on accountability plans in Connecti- 
cut, Maine and Massachusetts to provide current 
examples of the variation in the types of information 
gathered and the extent to which accountability is 



As community college 
practitioners often point 
out, not all students wish to 
attain degrees or certificates. 
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Selecting peer groups: An art and a science 



eer comparisons have become a popular component of performance accountability 
systems and assessment practices. But how are peer groups identified? Peer selection 
often combines objective data analysis and political wrangling, as college leaders seek 
to position their institution well in relation to the peer group for subsequent performance 
reports. Administrators, institutional researchers, state system analysts and external consultants 
are among those who get involved in identifying a group of potential peers. During this 
process, administrative leaders, in particular, watch for potentially negative funding and public 
relations implications. Throughout this process, each institution remains aware of its unique 
characteristics and those of the communities and students it serves. It is likely no peer group 
will ever be considered perfect by all interested parties, especially when comparisons are used 
to mark some institutions as "underperforming." 

For those navigating the political waters of peer group selection, several resources describing 
the "science" of selection processes are available. The most comprehensive treatment is 
provided by a recent entry in the New Directions for Community Colleges series, available 
from ]ossey Bass Publishers. Alexander McCormick and Rebecca Cox (2003) edited 
Classification Systems for Two-Year Colleges, which includes chapters considering groupings 
based on institutional characteristics, curricular emphases and market orientation. The 
Community College Review also recently published an article by Rodney Hurley (2002) that 
reviews the types and purposes of community college peer groups and the analytical strategies 
for creating them. Though focused on the four-year sector, an article by Sharon Weeks and 
colleagues (Weeks, Pucket SC Daron, 2000) that appeared in Research in Higher Education 
explicitly tackles the tension between politics and objective analysis and describes the authors' 
efforts to find a meaningful balance between the two. 



truly focused on performance outcomes. In Con- 
necticut and Massachusetts, performance reports 
are required as part of the budgeting process. The 
accountability objectives in Maine were devel- 
oped through a voluntary effort on the part of 
the colleges in response to a perceived need to be 
proactive in providing public accountability. Table 
2 provides a summary of performance reporting 
standards from public documents available on the 
state system Web sites in Fall 2004 . It is important 
to note that this summary is a "snapshot" from a 
particular point in time. Standards often change 
from year to year within a state, and a gap often 
exists between formally adopted standards and 
those in use. The three states differ in the extent 



to which they present their accountability plan in 
terms of objectives or indicators. Thematic titles 
or groups of indicators have been retained in Table 
2 where presented by the state, but the groupings 
have been reordered for cross-state comparison. 

The performance accountability movement 
was intended to highlight institutional results to 
motivate increased productivity. As expected, the 
performance-reporting standards in Table 2 include 
many outcome indicators. These include the ex- 
pected measures of student persistence, certifica- 
tion, transfer and graduation rates, which are found 
in Section 1 of the table. Maine and Massachusetts 
also seek to measure student satisfaction, reflecting 
the desire to include indicators that judge commu- 
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nity college effectiveness based on colleges' ability 
to assist students in meeting their goals, which do 
not always include obtaining a credential. All three 
states also include enrollment among the reporting 
standards (as shown in Section 2 and in Section 3), 
which emphasizes the numbers of students being 
prepared in occupational fields. Occupational per- 
formance outcomes such as employment rates also 
are included in Section 3. 

The cost-effectiveness 
of different types of higher 
educational programs and 
administrative practices has 
rarely been studied in rigor- 
ous terms. The performance- 
reporting standards from 
these three states suggest 
movement in that direction 
through categories of indica- 
tors intended to measure re- 
source efficiency, as indicated in Section 4. These 
standards appear to be in their infancy because 
they attempt to measure complex relationships be- 
tween resource use and student outcomes at high 
levels of aggregation. For example, in a category 
of indicators called "resource efficiency," Connecti- 
cut reports operating expenditures per student 
in conjunction with graduation rates. Similarly, 
Massachusetts reports the percentage of educa- 
tional and general expenditures in administrative 
areas, with an eye toward keeping administrative 
spending low. Neither strategy offers great prom- 
ise for an understanding of the most effective use 
of available resources to achieve optimal student 
outcomes because the gross measures of inputs 
and outputs provide no information about actual 
resource use for a wide variety of instructional and 
administrative activities. Maine states the objective 
of observing and reporting "efficient utilization" 
of funds but does not offer ways to measure such 
efficiency. 

The NCCBP collection of information about 
class sizes, student/faculty ratios, instructional 
faculty loads and training expenditures, along with 
fine-grained student outcome indicators at the level 



of course type, is preferable to these state-level 
attempts at measuring efficiency. Ffiowever, even 
for the NCCBP participants, the challenge remains 
to link resources and student outcomes within par- 
ticular curricular and programmatic areas. These 
shortcomings mainly illustrate the limitations of 
performance benchmarking, which may call for 
reports of inputs and outputs but does not offer a 
mechanism for understanding how resources are 
used effectively. That is the 
task of the admittedly more 
complicated and expensive 
form of peer comparisons 
called process benchmarking, 
which is discussed below. 

The development of 
feasible strategies to measure 
institutional cost-effectiveness 
would mitigate the strong and 
legitimate objection often 
raised by practitioners to performance account- 
ability: It is not fair to hold institutions account- 
able for achieving equal outcomes with unequal 
resources. This is particularly true when colleges 
enroll students with varying developmental needs 
and when resource disparities among colleges 
in the same state are often quite large (Dowd & 
Grant, forthcoming-a, forthcoming-b). 

The final two sections of Table 2 include 
indicators of instructional design and collabora- 
tion within higher education and between higher 
education and high schools. These indicators focus 
on processes that are expected to enhance perfor- 
mance. The inclusion of narrative descriptions as 
part of the reporting requirements highlights the 
difficulty of reducing all forms of valued activities 
to countable pieces of evidence. Maine, in particu- 
lar, with its voluntary reporting system, establishes 
the use of data collection, analysis and report- 
ing processes as valuable in and of themselves to 
inform performance-enhancement goals. 

Notably, Maine also calls for a performance 
standard of expenditures on professional develop- 
ment. This standard, which was set in 2004 at 2 
percent of each college's operating budget, serves 



It is not fair to hold 
institutions accountable for 
achieving equal outcomes 
with unequal resources. 
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as a recognition (unusual in accountability plans) 
that education is a complex endeavor and that 
achieving higher performance will require higher 
levels of professional knowledge and training. 
Professional development activities appear to be 
chronically underfunded in community college 
systems, with many institutional researchers, fac- 
ulty members and administrators receiving minimal 
funding, or even no funds, for travel to profes- 
sional conferences. 

To improve institutional performance, Maine's 
voluntary reporting strategy may well be the 
best approach. Scholars have argued that it is 
simply not possible to impose accountability 
(Koestenbaum & Block, 2001), particularly in high- 
pressure public and political arenas. The moment 



an institution's weaknesses are to be exposed 
publicly, numerous organizational defenses will 
be stimulated to deflect criticism rather than to 
undertake real reform (Dowd & Tong, 2004). 
Connecticut's accountability system presents a 
best-practice strategy in this regard. Public reports 
group results by categories of institutional size 
that include at least three colleges. Academic 
studies involving practitioners show that 
administrators and faculty who engage in in-depth 
data analysis often become agents of change 
on their campuses when the inquiry process 
involves them in deciding which student outcome 
indicators to examine (Bensimon, 2004,- Bensimon, 
Polkinghorne, Bauman & Vallejo, 2004). 
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Diagnostic benchmarking 



T o address the limitations of per- 
formance benchmarking to inform 
understanding of the processes that 
influence student outcomes, many 
community colleges have begun to 
adopt assessment instruments and procedures 
marketed by several national organizations. 
These assessments center on surveys of student 
attitudes and behaviors, as 
well as their satisfaction 
with various aspects of the 
collegiate experience. As 
the same questionnaires are 
adopted by peer institutions 
nationwide, the results cre- 
ate national databases and 
provide a resource for institu- 
tions to conduct diagnostic 
benchmarking. (See Page 15: 

"Adopting a national survey 
or assessment instrument?") 

The diagnostic checks and recommended institu- 
tional review procedures vary with each assess- 
ment instrument, but each offers an explanatory 
framework for analyzing results and planning for 
institutional improvement. By adopting a nation- 



ally available survey rather than designing their 
own, colleges can then compare their institutional 
results to national norms and engage in strategic 
planning to improve their practice. 

National surveys of student attitudes, experi- 
ences and behaviors that provide benchmarking 
data for participating colleges include the Commu- 
nity College Student Experiences Questionnaire 
(CCSEQ), the Community 
College Survey of Student 
Engagement (CCSSE), the 
Cooperative Institutional 
Research Program (CIRP), 
the Faces of the Future survey 
from AACC in collabora- 
tion with ACT, the Student 
Opinion Survey from ACT, 
and the Noel-Levitz Student 
Satisfaction Inventory (SSI). 
The models and measure- 
ment characteristics underly- 
ing these surveys are summarized in Table 3 (Pages 
28-30). 

The designs of the first three surveys presented 
in Table 3, the CCSEQ, the CCSSE and the CIRP, 
are based on educational research and theories. 



Community colleges have 
begun to adopt assessment 
instruments and procedures 
marketed by several 
national organizations. 
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Adopting a national survey 

OR ASSESSMENT INSTRUMENT? 



he American Council on Education in collaboration with the Association for 
Institutional Research (AIR) published a comprehensive guide to nationally available 
surveys and assessment tools in 2001. Measuring Quality: Choosing Among Surveys and 
Other Assessments of College Quality by Victor M. H. Borden and ]ody L. Zak Owens provides 
a thoughtful decision guide and a table summarizing report information, availability of 
national comparison data, options for adding local questions and subscribing as a consortium, 
cost and complete contact information. The table includes assessments of student outcomes 
and academic proficiencies, as well as of faculty, administrators and alumni. It includes 
assessment tools for all sectors of higher education, but instruments designed specifically for 
community colleges and their students are well represented. 

Though many assessment instruments have evolved since the guide's publication in 2001, the 
classification of survey types and the decision framework are invaluable. Characterizing the as- 
sessments as "information tools for both accountability and improvement," the authors set out 
three general criteria for determining their usefulness: "The appropriateness of the tool for the 
specific job at hand; the skills and experiences of users; and the availability of sufficient finan- 
cial, personal, and material resources" (p. 3). 

The guide is available for free download at the AIR Web site: (http: / / www.airweb.org/). 



Student outcomes are conceptualized as result- 
ing from the student's experience in the college 
environment. The student-college relationship 
is interactive. Students are not passive "widgets" 
processed in an educational factory. Students 
themselves contribute to their own outcomes. 

The student role is characterized by the CCSEQ 
in terms of "effort" and by the CCSSE in terms of 
"engagement." The CIRP focuses on broad indica- 
tors of students' experiences in college, as well as 
their pre-collegiate expectations. 

Engagement, as measured in the CCSSE, is 
not a single construct. It is a family of five types of 
indicator categories, or benchmarks, drawn from 
studies of student learning and persistence that are 
thought to influence a student's academic success 
(Engagement by Design, 2004,- Marti, 2004). As in 
the CCSEQ, these include student effort as one 



of five benchmarks. Among the other four CCSSE 
benchmarks, three are determined interactively by 
student behaviors and the college environment. 
These are active and collaborative learning, the 
level of academic challenge, and student- faculty 
interaction. The college clearly has a role to play 
in these areas in creating effective learning envi- 
ronments to facilitate student effort. The fifth cat- 
egory of indicators, support for learners, represents 
an area that is the responsibility of the college. A 
college's survey results serve as diagnostic indica- 
tors of areas of strength or needed improvement. 
For example, a college observing relatively low 
levels of student-faculty interaction might adopt 
policies concerning faculty advising, structured 
orientations or curriculum changes. 

The remaining three surveys in Table 3 — the 
Faces of the Future survey (FOF), ACT Student 
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Communities of practice effecting change 



ow will collecting data about student outcomes contribute to effective educational 
practices? Research conducted as part of the Diversity Scorecard Project at the University 
of Southern California's Center for Urban Education suggests that it is essential to involve 
campus practitioners in defining problems of student success (Bensimon, et al., 2004). The 
Diversity Scorecard Project, which is directed by Estela Mara Bensimon, has been implemented 
in 21 two- and four-year public and private colleges in six states. It is also the foundation for a 
new project, Equity for All: Institutional Responsibility for Student Success, which is funded by 
Lumina Foundation and the Office of the Chancellor for the California Community Colleges and 
will involve 10 community colleges in that state. 



Writing in Change magazine, Bensimon (2004) explains that the Diversity Scorecard is a 
framework for institutional self-assessment that starts with campus teams of administrators and 
institutional researchers examining student outcome data disaggregated by race and ethnicity. The 
team then draws conclusions about inequities in student outcomes and establishes performance 
goals for addressing the problems they have identified. Team members often then take on the 
role of change agents in earnest. The Diversity Scorecard Project illustrates that when faculty, 
administrators and institutional researchers generate knowledge of problems on their own 
campus, their engagement in the process of knowledge creation contributes to their openness to 
arguments for change and their willingness to advocate for new policies and practices. 



The strategy of the Diversity Scorecard and Equity for All projects is to help faculty, 
administrators and researchers act "self-reflectively and collaboratively within everyday practice" 
(Bensimon et al., 2004, p. 109). In this way, practitioners are at the center of data interpretation 
for the purpose of informing knowledge, beliefs and actions. Organizational learning takes place 
within a community of practice that has the capacity to foster student success. 



The inquiry teams meet several times to examine fine-grained institutional data on student 
outcomes. These analyses provide the basis for benchmarking institutional improvement on four 
sets of "vital signs" that function as diagnostic indicators: academic pathways (e.g., vocational or 
academic tracks), retention and persistence, transfer readiness, and excellence (e.g., enrollment in 
competitive majors or honors programs). 



Each college chooses its own set of indicators and benchmark goals. However, the project 
provides a structured assessment process that is shared among participating institutions. Colleges 
complete an Equity Scorecard and present their findings in terms of a standardized Equity Index, 
a ratio of the percentage of students in different racial and ethnic groups attaining specified 
educational outcomes and their percentage in the relevant community population. 



The Equity Scorecard and the Equity Index provide a standardized benchmarking process and 
indicators, both of which facilitate cross-campus comparison about the assessment processes that 
support organizational change and help reduce inequalities in educational attainment. 
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Opinion Survey (SOS) and the Noel-Levitz Stu- 
dent Satisfaction Inventory (SSI) — in addition to 
noting student characteristics and expectations, 
focus on student satisfaction across the gamut of 
collegiate administrative functions. The SSI, for 
example, measures satisfaction with academic 
advising, instruction, safety, registration, aca- 
demic support and other functions, for a total of 
12 areas of student concern. Checking the health 
of an organization against a series of "dashboard" 
indicators of this type is the hallmark of diagnostic 
benchmarking. With this type of assessment, col- 
leges can maintain or improve a consumer service 
orientation and allocate resources to improve 
weaknesses. Particularly when paired with the use 
of surveys based in educational theory, surveys of 
student satisfaction can serve as valuable manage- 
ment tools for improving campus functions. On 
their own, though, they are unlikely to provide 
insight into steps that may be necessary to educate 
students more effectively or to raise students' expecta- 
tions for their own performance. (For another type 
of diagnostic benchmarking, see "Communities of 
practice effecting change," Page 16.) 

The growing use of such uniform, profession- 
ally designed and theoretically contextualized 
surveys is part of the growing sophistication of 
higher education benchmarking. When grounded 
in educational research, these indicators provide 
explanatory frameworks for college student out- 
comes in ways that performance measures, which 
grow out of business and policy perspectives, do 
not. As shown in Table 3, tests of reliability for 
questions on five of six of the surveys are readily 
available. The generally high scores on the statis- 
tic representing the consistency of respondent an- 
swers (Cronbach's alpha coefficient) demonstrates 
that the questions are understood by respondents 
and are well designed. Tests of validity, which 
indicate whether the questions measure what is in- 
tended and whether those measures are relevant to 
the ultimate outcomes of interest, are also readily 
available for four of six surveys. Validity is trickier 
to establish, and the survey designers draw on the 
educational literature, interviews with students 



and expert reviewers, and multivariate statistics for 
this purpose (Ethington, Guthrie & Lehman, 1999,- 
Ethington & Polizzi, 1996,- Marti, 2004,- Ouimet, 
Bunnage, Carini, Kuh & Kennedy, 2004,- Pace, 1999). 

Though the surveys are professionally de- 
signed and commercially marketed, their adop- 
tion on campuses does not reduce the burden on 
administrators and institutional researchers to 
understand the technical aspects of the question- 
naire design. (See Page 18: "Designing a survey?") 
On the contrary, it may increase the need for pro- 
fessional education and development in this area 
so decisions about which survey to administer at 
different points of an institution's assessment and 
planning cycle are well informed. 

One limitation of these surveys to inform 
understanding of institutional performance and ef- 
fectiveness is suggested by studies of institutional 
climate, student identity development and cultural 
adaptation in educational settings (Dowd & Korn, 
2005). The conceptual frameworks underlying 
surveys of student attitudes and behavior are not 
strongly informed by studies of campus climate, 
which emphasize issues of cultural difference, 
conflict, dominance and discrimination. The theo- 
ries of attrition and persistence that informed the 
development of the scales to measure "effort" and 
"engagement" (Astin, 1993,- Pascarella & Teren- 
zini, 1991,- Tmto, 1987) are now being revised by 
a new generation of scholars who emphasize the 
demands on students of acting as "cultural workers" 
to create spaces on the campus where they feel at 
home (Gonzalez, 2001) and of developing bicul- 
tural identities and networks (Nora, 2001-2002,- 
Rendon, Jalomo & Nora, 2000). 

The importance of student effort to maintain 
supportive relationships within one's own cultural 
affinity group, which may be termed "cultural ef- 
fort" (Dowd & Korn, 2005), was not envisioned by 
the theoretical underpinnings of student-institu- 
tion fit models, so it is not measured. For example, 
questions measure students' efforts to interact with 
faculty and with students whose cultural back- 
grounds differ from their own. Theories of cultural 
identity and student development suggest the need 
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for additional items that measure students' efforts 
to maintain strong ties with their families and 
home communities, as well as with other students 
on campus who share their cultural heritage. 

Though the literature suggests that feelings of 
marginalization and discrimination would certainly 
contribute to student attrition, these concepts are 
not measured in national instruments. Colleges can 
pursue two strategies to better understand these 
issues. 

The first strategy is to conduct focus groups 
with students to explore the effects of campus 
climate, services and curriculum on students' feel- 
ings of belonging or alienation on campus. Focus 
groups provide the opportunity to explore factors 
not captured on standardized measures and to 
customize inquiry to specific institutional and com- 
munity contexts. 

The second strategy is the design of local 
surveys by institutional researchers in collabora- 
tion with administrators and faculty. The items 
for inclusion can also be identified through the 
focus groups and by pilot testing of the survey on 
campus. The results of such efforts would suggest 
new avenues for college performance in regard 
to facilitating community connections and family 
involvement on campus. 



Such pairings of focus groups with the admin- 
istration of local and national surveys are likely to 
emerge as a best practice for assessment as bench- 
marking practices continue to evolve. In conducting 
focus groups, however, administrators and institutional 
researchers must be mindful of their role as college 
"insiders" (Banks, 1998) and actively structure focus 
groups in ways that will generate "outsider" per- 
spectives. The creation of consortia of institutional 
researchers who can trade off in research roles on 
each other's campuses may be a cost-effective way 
to avoid insider bias. Focus group research of this 
type will likely contribute to the development of 
the next generation of national surveys. 

Three types of comparisons are available to 
colleges that administer national surveys. The first 
is comparison of institutional scores to national 
norms. The second is comparison of institutional 
scores to those of a smaller number of peer institu- 
tions. The third is comparison of the portrait that 
emerges of college priorities and practices against 
the institution's stated mission. These comparisons 
may alert campuses to issues needing attention. 
Fdowever, to motivate institutional change and 
promote effective practices, they must also be part 
of in-depth discussions to frame understanding of 
the nature of the problems to be addressed. 



Designing a survey? 



primer describing survey design and administration techniques called What Is a Survey? 
is available free of charge on the Web at: http:/ / www.whatisasurvey.info/. Authored 
by Fritz Scheuren (2004), who edited a series by the same name for the American 
Statistical Association, this resource includes chapters on survey planning, collection and 
quality. It also describes the value of complementary strategies such as focus groups and 
cognitive pre-test interviewing. The guide is written in accessible language suitable for non- 
specialists. 

The Association for Institutional Research (http:/ / www.airweb.org/), through its Resources for 
Institutional Research series, has available for purchase Questionnaire Survey Research: What Works 
(1996) by Linda A. Suskie. This comprehensive guide includes survey planning and design topics, 
as well as discussion of statistical reliability, validity and data analysis. It is written in a clear, 
concise manner with specific applications to higher education contexts. 
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Process benchmarking 



P rocess benchmarking has been relatively 
absent from accountability plans because 
of its greater expense and time require- 
ments, but interest in this approach has 
grown recently. In keeping with the 
federal government's stated preference for funding 
experimental and quasi-experimental research to 
identify effective educational practices ( Rigorous 
Evidence; Scientifically-Based 
Research), researchers have 
recently proposed the de- 
velopment and adoption of 
educational experiments, or 
"regimes," to assess institu- 
tional productivity. These ex- 
periments would be defined 
in terms of three "critical 
features": (1) outcome mea- 
sures,- (2) descriptions of the 
optimal academic "treatment" 
conditions, including the 
tasks undertaken and instructional media employed 
in instruction, and (3) descriptions of the optimal 
teaching strategies associated with those tasks 
and technologies adopted to produce the speci- 
fied outcomes (Cohen, Raudenbush & Ball, 2003, 



p. 1 35). Such experiments are a form of process 
benchmarking because organizations participating 
in them would have the opportunity to observe 
how variations in teaching or student service meth 
ods affect student outcomes. The examination of 
the "production" processes of education under this 
type of process benchmarking is intended to be 
in-depth and precise. 

Because the federal gov- 
ernment now favors the use 
of experimental designs for 
the evaluation of federally 
funded programs such as 
TRIO and GEAR Up, com- 
munity college faculty and 
administrators can expect to 
be invited with increasing 
frequency to participate in 
experiments designed to test 
the effectiveness of different 
educational and administra- 
tive strategies. This type of process benchmarking 
dovetails with the assessment movement. Through 
their internal assessment efforts, community col- 
lege educators and researchers have already begun 
to closely define learning outcomes, performance 



Researchers have recently 
proposed development of 
educational experiments, or 
"regimes," to assess 
institutional productivity. 
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requirements and competencies (An Assessment 
Framework, 2004). (See below: "Assessment in the 
learning college.") Participation in experimental 
regimes will, nevertheless, present challenges, par- 
ticularly due to the high level of human resources 
required for sustained participation and collabo- 



ration with faculty and administrators in other 
colleges. While assessment efforts are generally 
focused internally as a college effort, experimental 
evaluation designs are likely to require colleges to 
make detailed comparisons of their educational 
practices and outcomes with other institutions. 



Assessment in the learning college 



ong before the current interest in data-driven decision-making, community college leaders 
were actively defining and assessing learning environments that foster student success. The 
concept of learning-centered colleges has generated important assessment principles for 
observing student experiences and outcomes [Barr SC Tagg, 1995; Chickering SC Gamson, 1987; 
Ewell, 1991, 1997; O'Banion, 1997, 1999; Tagg, 2003). 

In fact, as characterized by Kay McCIenney [2003] in a recent article in the American 
Association for Higher Education's Inquiry 8C Action, assessment practices are among the 
defining characteristics of learning-centered colleges. In the article, McCIenney highlights six 
key characteristics — each paired with a rationale for its importance and examples of systems, 
processes, policies and practices — that serve as evidence of being learning-centered: 

1. The institution has clearly defined outcomes for student learning. 

2. The institution systematically assesses and documents student learning. 

3. Students participate in a diverse array of engaging learning experiences aligned with 
required outcomes and designed in accordance with good educational practices. 

4. Data about student learning typically prompt — and support — the institution and 
individuals to reflect and act. 

5. The institution emphasizes student learning in its processes for recruiting, hiring, 
orienting, deploying, evaluating and developing personnel. 

6. Key institutional documents and policies, collegial effort and leadership behavior 
consistently reflect a focus on learning. 

The League for Innovation in the Community College also recently issued a resource guide 
describing the principles of a "learning college" and providing examples of their application. 

"An Assessment Framework for the Community College: Measuring Student Learning and 
Achievement as a Means of Demonstrating Institutional Effectiveness" [An Assessment Framework, 
2004] is available at: http:/ / www.Ieague.org/ publication /whitepapers/ 0804.html. 

The guide stresses the importance of a shared assessment vocabulary among stakeholders so that 
meaningful discussions of assessment implementation may take place. It also provides an eight- 
step process for developing and implementing assessments. This process is illustrated by case 
studies chronicling the problems faced by four community colleges, how each college developed 
and implemented assessment strategies reflective of institutional distinctiveness, and the results 
of their efforts. 



CULTURE OF INQUIRY 20 




Assessment in the learning college (continued) 

The following steps for the development and implementation of assessments are reproduced from 
pp. 16-17 of the League's framework: 

1. Define measurable institutional learning outcomes. Establish outcomes at the institutional, 
program, major, course and classroom levels. 

2. Design assessments to measure learning outcomes. Determine the outcomes to measure; 
determine the purpose for the assessment; determine the assessment method to employ, 
and determine the kind of assessment data you need to collect. 

3. Design learning events based on learning outcomes. Include assessment activities within 
the learning designs. 

4. Deliver learning. 

5. Assess learning and learning events. 

6. Gather and format data generated from assessment activities. 

7. Interpret the assessment data. 

8. Use assessment data to make decisions at the student, classroom, course, major, program 
or institutional level. 

Along with this holistic view of colleges as learning-centered has come a need for organizational 
structures that integrate teaching and learning in and out of classrooms. Many community 
colleges have begun reorganizing to become "learning-centered." Today, with learning paradigm 
concepts part of our everyday language of higher education, the challenge is to measure and 
demonstrate the effectiveness of these new organizational structures in promoting holistic 
student learning. 

in a recent working paper, Esposito and Dowd [2004] describe the reorganization of Paradise 
Valley Community College [PVCC] in Arizona to illustrate the significant, long-term institutional 
effort necessary to become a learning-centered institution. The elements of an institutional com- 
mitment to transformation are illustrated by the case of PVCC, one of 10 Maricopa County 
Community Colleges that were pioneering colleges in the larger movement toward a learning 
paradigm. The report is available at: 

http: / / www.facuity.umb.edu / aiicia_dowd/ ccssp/ working_reports.htm. 



Note: The text for this sidebar was prepared by research associate Linda Kent Davis. 
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Discussion 



B enchmarking practices in higher educa- 
tion have evolved and are becoming 
increasingly sophisticated. Administra- 
tors and faculty members have numerous 
options regarding the kinds of activities 
to participate in and support on their campuses. 
Some participation is mandated by state and fed- 
eral governments. To inform the development of 
these mandates, practitioners 
will benefit from knowl- 
edge of the broad array of 
benchmarking practices now 
in use across the country and, 
as provided in this report, 
a language with which to 
distinguish the purposes of 
these practices. Benchmark- 
ing activities will be most 
beneficial to those who 
understand the underlying 
theoretical frameworks and 
statistical foundations of various approaches and 
can interpret results in the appropriate context. 
The task of gaining this knowledge certainly cre- 
ates a greater need for professional development, 
training and education. 



If peer comparison processes are to spur 
innovation, the results must inform the attitudes 
and behaviors of faculty and administrators 
regarding the best strategies for ensuring student 
success. The notion of a culture of inquiry puts 
practitioners at the center of change efforts. (See 
Page 16: "Communities of practice effecting 
change.") As sociologist Richard Alford emphasizes 
in his engaging text The 
Craft of Inquiry, "Evidence 
never contains its own 
explanation" (1998, p.29). 
Evidence does not stand 
independently as true or 
meaningful. Recognizing the 
craftsmanship in the inquiry 
process entails recognition 
of the personal perspectives 
and experiences we all 
bring to the interpretation 
of evidence. It means that 
the preparation and process for deciding what 
information to collect, whom to involve in data 
interpretation, and how to communicate results are 
as important as the results themselves. This is the 
critical difference between a culture of evidence 



If peer comparison 
processes are to spur 
innovation, the results 
must inform attitudes 
and behaviors. 
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and a culture of inquiry: The emphasis shifts from 
the data to the decision-maker as the locus of change. 

The following questions, which highlight the 
importance of the decision process, are ones ad- 
ministrators should ask before embarking on peer 
comparison activities: 

1 . How well does the benchmarking strategy 
match my understanding of the problem? 

2. Will the results be compared against an ap- 
propriate comparison group? 

3. Will the strategy inform my understanding 
of the problem? 

4. Will new understanding of the problem 
provide insights for solutions? 

5. Will we be prepared and willing to make 
the changes indicated? 

6. How will we incorporate these insights into 
our practice? 

7. Will these solutions be consistent with our 
mission? 

8. Who needs to be involved in order to ef- 
fectively interpret and implement changes? 

An additional set of important questions con- 
cerns the relative costs and benefits of the variety 
of benchmarking options available. 

9. What are the direct costs of the bench 
marking project? 

10. What are the indirect costs of human re- 
sources to complete the activity and adopt 
solutions? 

1 1 . Will the services of an outside consultant 
be required to facilitate difficult reflective 
discussions about the nature of problems? 

12. What professional training and develop- 
ment is needed for those who participate in 
the activity or whose work is affected by 
adopted solutions? 

1 3. What kind of support, communication 
processes and resources will be needed 
to enable participants to sustain engage- 
ment in the project for its duration? 



1 4. What is the value of the anticipated benefits? 

These questions reflect the high degree of 
professionalism required for colleges to effectively 
engage in benchmarking activities with peer insti- 
tutions and sustain the inquiry process in ways that 
will support educational best practices. Questions 
10-13 focus on the human resource demands. The 
need for professional education and training in this 
era of high standards and accountability is clear. 
Whether adequate financial resources have been 
allocated by states and colleges to fund ongoing 
professional development is not clear. Maine is one 
state that sets a performance-reporting standard 
for professional-development expenditures. Is 2 
percent of a community college's operating budget 
an adequate level of expenditure for professional 
development? The question deserves further at- 
tention. How does this level of spending compare 
with typical expenditures for professional devel- 
opment and training in the business sector or in 
private research universities? 

There is widespread recognition of the chal- 
lenges community colleges face in educating di- 
versely prepared students to achieve a wide range 
of educational goals. Enthusiasm for accountabil- 
ity plans that function through "carrot and stick" 
strategies is due, thanks to a growing sense of their 
ineffectiveness (Accountability for Better Results, 
2005). During the era of accountability, the tools 
and strategies for benchmarking institutional 
performance and effectiveness have evolved and 
grown increasingly sophisticated. These develop- 
ments owe as much to the collegiate assessment 
movement, which has focused independently on 
characterizing good teaching and learning envi- 
ronments, as to accountability pressures. A new 
era of accountability will hold greater promise for 
informing effective educational practice if it incor- 
porates respect for the professionalism and profes- 
sional development needs of community college 
administrators and faculty as a central tenet. 
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Table 1: National Community College Benchmark Project 
performance indicators and peer-selection criteria 

Outcome indicators 



Certificate, degree completion and transfer rates. 

Credit course persistence and success rates. 

Performance in transfer institutions. 

Student satisfaction ratings. 

Student goal attainment. 

College-level course retention and success rates. 

Developmental course retention and success rates. 

Developmental student success in first college-level courses. 

Business and industry courses (includes vocational training and professional 
education at business sites). 

Career program completers' employment status and employer ratings. 

Success rates in core academic skill areas. 

Institution-wide grade distribution. 

Input indicators 

Enrollment rate: the proportion of the service area population enrolling in 
credit or non-credit courses. 

Community participation rate: the proportion of the service area population participating in 
cultural activities, public meetings and sporting events on campus. 

Minority student participation rates. 

Service area high school graduate enrollment rates. 

Average credit section size. 

Student/faculty ratio. 

Instructional faculty load. 

Expenditures per credit hour and FTE student. 

Student/student services staff ratio. 

Training expenditures per employee. 

Peer-selection criteria 



Unrestricted operating budget. 

Percent operating revenue from state. 

Faculty unionization. 

Service area population. 

Service area unemployment rate. 

Service area median household income. 

Service area percent traditionally underrepresented minority students. 

Source: The National Community College Benchmarking Project, www.nccbp.org. This is a partial list of performance indicators and 
peer-selection criteria,- for a full listing and definitions, see the glossary of data elements on the project Web site. 
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Table 2: Community college accountability objectives and 
performance indicators: three states 



Indicator category State Objectives and/or categories of indicators 



1 . Student 
outcomes 


Connecticut 


License and certification exam performance,- transfer to four-year 
colleges and into community colleges. 

Five indicator categories, including general enrollment. 


Maine 


Achieve standards of quality in core services of the System and colleges. 
Four indicators regarding student satisfaction with core services, graduate 
certification and licensure, faculty and staff professional development. 


Massachusetts 


Improve student access and achievement. 

Four indicators, including credit course completion rates, first-year retention rates, 
number of degrees and workforce certificates awarded, and nursing exam pass rates. 

Recruit and enroll qualified students. Provide high-quality learning 
opportunities. 

Three indicators of student satisfaction regarding Quality of learning experience, 
workforce preparation, and academic preparation for four-year sector. 


2. Enrollment, 

access, 

affordability 


Connecticut 


Minority enrollment,- operation expenditures from state support,- real 
price to students. 

Three indicator categories, including appropriations for student financial aid. 


Maine 


Increase enrollment to at least 15,000 matriculated students through 
increased capacity, accessibility and affordability. 

One indicator for non-matriculated enrollment growth and two categories of 
process requirements for goal-setting, data-gathering and reporting regarding 
student retention. 

Continue to foster and enhance an environment that promotes diversity 
One process requirement for assessing diversity, setting diversity goals, and 
monitoring progress. 


Massachusetts 


Ensure accessibility to affordable higher education for all residents of the 
Commonwealth. 

Six indicators, including enrollment, minority enrollment, tuition and fee charges 
relative to median income and total revenues, and financial aid awards. 


3. Economic 
and workforce 
development 


Connecticut 


Degrees conferred by credit program,- reports to the Connecticut 
Employment and Training Commission report card on employment and 
training program. 

Two indicator categories, including employment at graduation and six months 
later. 
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Table 2: Community college accountability objectives and 
performance indicators: three states (continued) 



Indicator category State Objectives and/or categories of indicators 





Maine 


Be an educational leader in economic development and an innovative 
contributor to economic growth. 

One category of indicators regarding credit, non-credit and contract training 
enrollments serving business and industry. 

Complete the transition to comprehensive two-year community colleges. 
Two indicator categories regarding the proportion of occupational programs 
offered and maintenance of program standards and one process requirement 
regarding monitoring of course offerings. 


Massachusetts 


Respond to specific needs of the workplace. 

Five indicators of workforce development enrollment, workforce placement, cost- 
sharing with business and industry, and transfer of workforce training students 
to the four-year sector. 


4. Resource 
efficiency 


Connecticut 


Real cost per student,- retention rates,- graduation rate,- enrollment by 
credit program. 

Four categories of indicators. 


Maine 


Assure that the community colleges have adequate financial resources to 
fulfill their mission by securing and maintaining increased state 
appropriations and federal, private and alternative funding, and by the 
efficient utilization of such funding. 

Two indicators of investments in capital assets and a process requirement for 
monitoring capital planning and budgeting. 


Massachusetts 


Ensure cost-effective use of resources and manage campuses efficiently. 
Three indicators of resource allocation to capital renewal, administration and 
other expenditure categories, and one process requirement for a financial audit. 


5. Collaboration 


Connecticut 


Collaborative activities with public schools. 

No indicators. Report evidence of activities and K-i 2 program support. 

Responsiveness to societal needs. 

Non-credit instruction,- collaborative activities within the community. 
One category of indicators, including personal and workforce development, and 
a category requiring narrative descriptions of collaboration. 


Massachusetts 


Promote collaboration among the campuses and with private higher 
education. 

Two indicators of cost-sharing with other institutions and student transfer. 


6. Instructional 
design 


Maine 


Use technology-mediated delivery methods that best serve the evolving 
needs of students. 

Two indicators of program delivery through computer technology and online 
course offerings. 
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Table 3: Comparison of surveys of community college students' 
experiences, behavior and attitudes 

SURVEY Community College Student Experience Community College Survey of 

Questionnaire (CCSEQ) Student Engagement (CCSSE) 



Focus 


Focuses on the quality of effort expended by 
students in academic areas, including in and 
out of class. Also assesses progress toward 
education goals, satisfaction with environment, 
and collects demographic and background 
information. (CSEQ is the alternate form for 
four-year colleges.) 


Focuses on institutional practices and 
student behaviors that research associates 
with positive collegiate outcomes 
("engagement"). (NSSE is the alternate form 
for four-year colleges.) 


Theoretical 

foundation 


Based on the observation that student effort 
affects academic achievement, satisfaction and 
persistence. 


The concept of engagement is derived from 
student persistence and learning theory. 
(The theoretical foundation for each 
question on the survey is presented by a 
link on the CCSSE Web site.) 


Measurement 


Measures quality of effort in eight distinct 
areas of college activity, based on multi- 
item scales. Each item in the scale measures 
activities that require incrementally greater 
effort. Also measures student gains on 25 
desired learning and personal development 
outcomes. 

Sampling: Purposeful sampling stratified by 
class type and time. 


Measures five areas of collegiate 
experiences: active and collaborative 
learning, student effort, academic 
challenge, student-faculty interaction, 
support for learners. These five benchmark 
scales were constructed based on factor 
analysis and expert judgment of educational 
researchers. 

Sampling: administered in randomly 
selected classes, stratified by class time, as 
established by CCSSE staff. 


Benchmarks/ 
reporting of 
results 


The data are provided to the institution 
along with a descriptive statistical report of 
institutional results and the aggregated results 
for the annual population of participating 
colleges. Further data analysis is available for 
an additional fee. 

National aggregate scores are reported in the 
CCSEQ Test Manual and Comparative Data 
(Ethington, et al., 1999), based on an analysis 
of 18,000 students in more than 60 colleges. 


Results are provided for all CCSSE 
participants, different subgroups within the 
full population, and individual colleges. 
Results include national and institutional 
scores on the five benchmark scales. 
Reported annually in Engaging Community 
Colleges: National Benchmarks of Quality 
Summary. 


Reliability 
& validity 


Using Cronbach's alpha as a measure of 
internal consistency, the coefficients for eight 
scales measuring quality of effort in collegiate 
activities ranged from .82 to .93. Validity is 
established by the cohesiveness of constructs 
underlying the scales, as demonstrated by 
inter-item correlation and factor analysis. 


The reliability of the five benchmark scales, 
measured by Cronbach's alpha, range from 
.56 to .80. The validity of the benchmarks 
is established through factor analysis and 
regression models predicting student GPA 
and self-reported gains on four-point scales 
measuring academic, personal and career 
development. 


Sources 


http://coe.memphis.edu/CSHE/CCSEQ.asp 
(Ethington & Polizzi, 1996,- Pace, 1999) 


www.ccsse.org 
(Marti, 2004) 
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Table 3: Comparison of surveys of community college students' 
experiences, behavior and attitudes (continued) 



SURVEY Cooperative Institutional AACC/ACT Faces of the 

Research Program (CIRP) Future Survey (FOF) 



Focus 


Focuses on students' high school experiences, 
expectations, goals, reasons for attending 
college, and students' academic experience in 
college, behaviors and tendencies, and 
community involvement. 


Focuses on students' experiences in 
relation to their goals and expectations, 
including barriers faced by the student, 
employment background, reasons for 
enrollment, and the extent to which 
course-taking by the student has met the 
student's needs and expectations. 


Theoretical 

foundation 


Astin's theory of involvement, the I-E-0 model 
(student inputs, college environment and 
student outputs). 


Not available on ACT Web site. 


Measurement 


Measures self-reported college experiences, 
skill development during college, 
information about student profiles (sex, race, 
demographics, major), student expectations 
during college, attitudes and behaviors 
regarding leadership, self-reported academic 
abilities, attitudes and behaviors about 
volunteering. Uses a follow-up survey to 
assess engagement, academic achievement and 
interactions with peers and faculty. 

Sampling: Administered through convenience 
sampling in classrooms or other proctored 
settings, such as orientation. 


Measures demographic information 
(general, employment, education) and 
current college experiences, including 
access, purpose, learning, satisfaction, 
expected outcome, intent and transitions. 

Sampling: Both credit and non-credit 
students. 


Benchmarks/ 
reporting of 
results 


The American Freshman is the annual national 
report of CIRP norms on first-time, full-time 
freshmen. 


Analysis of national data is based 
on 48,763 student records from 157 
community colleges from the Fall sessions 
of the 2000-01 and 2001-02 academic 
years. Participating institutions receive 
national results, an individual report 
about their student populations, and a 
comparison of their data and national data. 


Reliability 
& validity 


The reliability of the survey scales are in the 
range of .85-. 95. 

Predictive validity draws on research published 
in How College Affects Students by Pascarella 
and Terenzini (1991). Concurrent validity 
established based on research published in 
What Matters in College: Four Critical Years 
Revisited by Alexander Astin (1993). 


Not available on ACT Web site. 


Sources 


www.gseis.ucla.edu/heri/cirp.html 


www.aacc.nche.edu 
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Table 3: Comparison of surveys of community college students' 
experiences, behavior and attitudes (continued) 



SURVEY ACT Student Opinion Survey (SOS) Noel-Levitz: Student Satisfaction 

Inventory (SSI) 



Focus 


Focuses on student satisfaction with and use of 
various college services, including academic, 
admission/registration, rules/policies, facilities and 
satisfaction with college environment. 

(An alternate four-year college form is available.) 


Focuses on student satisfaction with a wide range 
of college experiences and student perception 
of overall services of the college. Students 
report on importance to them of each college 
experience and level of satisfaction with service. 
(An alternate four-year college form is available.) 


Theoretical 

foundation 


Not available on ACT Web site. 


Principles of consumer satisfaction and 
organizational performance. 


Measurement 


Measures demographic and background items (age, 
race, sex, hours worked per week, educational 
goals, occupational plans). Measures satisfaction 
with college programs and services using a five- 
point Likert response scale. 

Sampling: Information provided only to colleges 
administering the survey. 


Uses 12 scales, including academic advising, 
campus climate, support services, concern 
for the individual, instructional effectiveness, 
admissions and financial aid effectiveness, 
registration effectiveness, responsiveness 
to diverse populations, safety and security, 
service excellence, student-centeredness 
and academic services. Seven-point Likert 
response scale assesses "importance to you," 
and "satisfaction with" various campus services. 
Three scores are then provided: an importance 
score, a satisfaction score and a gap score. The 
performance gap score indicates the difference 
between importance and satisfaction. Composite 
scales can be analyzed to determine trends in 
importance, satisfaction and performance over 
the most recent five years. 






Sampling: Administered in randomly selected 
classes, stratified by class time, selected by 
participating colleges. 


Benchmarks/ 
reporting of 
results 


Reports to participating institutions provide results 
as institutional statistics and graphs and national 
norms based on data from colleges that have 
administered the survey. Summary Report, Graphics 
Report, National Data Comparison Report and data 
diskette included in institutional survey fee. 


The 2004 National Satisfaction and Priorities 
Report provides benchmarks by and across 
institutional types from surveys administered Fall 
2001 through Spring 2004, including 259,493 
student respondents in 284 community colleges. 


Reliability 
& validity 


Overall reliability estimates of .92 for satisfaction 
with college services and programs (Section II) 
and .95 for satisfaction with various aspects of the 
college environment (Section III) are reported, 
which indicates high reliability. 

Validity: Not reported on Web site. 


Cronbach's alpha is .97 for the importance 
scale and .98 for the satisfaction scale, which 
demonstrates very high reliability. Three-week 
test-retest reliability is .85 for importance scores 
and .84 for satisfaction scores, demonstrating 
reliability over time. 

Validity: Convergent validity was assessed by 
correlating satisfaction scores from SSI with the 
College Student Satisfaction questionnaire (r= .7 1 ). 


Sources 


www.act.org 


www.noellevitz.com 
( National Student Satisfaction, 2004) 



Table 3 prepared by Diane Bourcfue, Alicia Dowd and Randi Korn 
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